Inspection system

ABSTRACT

A system for improving inspection systems is provided. The system generally includes a camera in communication with a processor. The camera is configured to capture an image for inspection of an object A to be inspected. The camera transfers the image to the processor for comparison with a reference image. If the comparison is not within a predetermined range, then the camera settings are adjusted and a new image is taken until the current image matches the reference image. Alternatively, or in addition to, the system is capable of learning based on a plurality of data and creating a model image for comparison prior to capturing the first image.

CROSS REFERENCE TO RELATED APPLICATIONS

The application claims priority and benefit to U.S. provisionalapplication Ser. No. 62/769,031 filed on Nov. 19, 2018, which isincorporated herein by reference.

TECHNICAL FIELD

The present specification generally relates to an inspection system forinspecting products on a production line. More specifically, the presentspecification relates an inspection system using a sensor, referenceimage and controlling sensor settings to optimize inspection of aspecific product on an assembly line.

BACKGROUND

Assembly line inspection systems using sensors are known in the art.However, currently known systems do not compensate for changes inenvironmental light surrounding the systems. If a building is lighted bynatural light, changes in day may disrupt the lighting of the sensor.These issues may arise of the inspection system is located near a dooror a window. Furthermore, if people or objects interfere with the pathof the ambient light within a building used to light an object to beinspected, problems with inspection may arise.

If a shadow is created on the object to be inspected, the light is lowand/or the light is too high, then the inspection system may give afalse positive or a false negative when inspecting the object. Forexample, if a person walks by the sensor at the moment an object isbeing inspected, then a shadow may be created on the object beinginspected. This could lead to a false negative of a defective part thusleading to the part being unnecessarily discarded.

Accordingly, a need exists for an improved inspection system.

SUMMARY

A system and corresponding flow chart depicting the improved inspectionsystem is described herein and along with the accompanying embodiments.The system generally includes a camera in communication with aprocessor. The camera is configured to capture an image for inspectionof an object A to be inspected. The camera transfers the image to theprocessor for comparison with a reference image. If the comparison isnot within a predetermined range, then the camera settings are adjustedand a new image is taken. Alternatively or in addition to, the system iscapable of learning based on a plurality of data and creating a modelimage for comparison prior to capturing the first image.

An automated inspection apparatus is described herein including a cameraconfigured to capture images of items, a processor in communication withthe camera, a histogram matching portion having a reference image, thereference image taken by the camera in optimal lighting conditions, thehistogram matching portion converting the image from RGB colorspace toHSV, wherein the camera taking a current image of the item to beinspected, the histogram matching portion converting the image from RGBto colorspace to HSV, the current image compared to the reference image,if the current image is equal to or within a predetermined range of thereference image, then the inspection proceeds forward, if the currentimage is not within the predetermined range of the reference image, thenthe image is retaken. The camera is positioned either above or adjacentto a conveyor belt so as to capture said images of said items.

Another embodiment provides for an automated inspection process usingmachine learning comprising the steps of storing and processing aplurality of data relating to a camera and/or image processing,generating by the camera a model image of an item to be inspected priorto taking a first image, determining if the model image is within apredetermined range, adjusting the camera settings to match a referenceimage, and taking a first image based on the model image. The pluralityof data may include prior camera setting data, weather data, lightingdata, time of day data, and/or third party data.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the subject matter defined by theclaims. The following detailed description of the illustrativeembodiments can be understood when read in conjunction with thefollowing drawings, where like structure is indicated with likereference numerals and in which:

FIG. 1 depicts a system diagram and corresponding flow chart of variousembodiments of the present inspection system according to one or moreembodiments shown and described herein

FIG. 2 depicts a generalized apparatus and assembly of the inspectionsystem with a downward facing camera according to one or moreembodiments shown and described herein; and

FIG. 3 depicts a generalized apparatus and assembly of the inspectionsystem with a side-facing camera according to one or more embodimentsshown and described herein.

DETAILED DESCRIPTION

FIG. 1 generally depicts a system and corresponding flow chart depictingthe improved inspection system of the present specification. The systemgenerally includes a camera in communication with a processor. Thecamera is configured to capture an image for inspection of an object Ato be inspected. The camera transfers the image to the processor forcomparison with a reference image. If the comparison is not within apredetermined range, then the camera settings are adjusted and a newimage is taken. Alternatively or in addition to, the system is capableof learning based on a plurality of data and creating a model image forcomparison prior to capturing the first image.

Referring now to FIG. 1, a system 200 is provided having both ahistogram matching approach and a machine learning approach, referredrespectively as 202, 204. The system 200 may have either or both ahistogram matching approach 202 and/or the machine learning 204. The twoapproaches may be used separately or together to enhance the results andare preferably used together if the time allotted for inspection allows.

The histogram matching 202 takes a reference image with the optimallighting conditions where the part being inspected is clearly visible.The histogram matching 202 then converts the image from the RGBcolorspace to HSV (hue, saturation, and value). The image is convertedto the HSV colorspace since it better isolates the intensity values ofthe image from the color components of the object being inspected. Oncethe intensity values have been identified, they are loaded into ahistogram and saved.

The same process is performed with the current image, which is to becompared to the reference image. The current image is converted to anHSV and the intensity values are loaded into a histogram. The histogramis then matched and compared to the reference histogram usingcorrelation as a metric. A perfect match between the reference image andthe current image is 1.0. If there is not a match, or if a match iswithin some predetermined threshold, the gain and exposure of the camera108 is modified until there is a good match.

As illustrated in FIG. 1, the histogram matching includes the processwhere a camera 108 takes an image of the object 106 to be inspected.Prior to inspection, the camera 108 takes an image, specifically thereference image, of the object 106 to be detected. During normalinspection, the same camera 108 takes the current image of the object106 to be inspected.

The camera 108 then communicates with the processor 112. The processor112 may be a computer or other device capable of processing andcomparing the reference image to the current image. The histogrammatching 202 includes the first step of comparing the current image tothe reference image. As discussed above, the current image and thereference image are both converted to HSV to better isolate theintensity values of the image from the color components. The histogrammatching then determines if the current image is within a predeterminedthreshold or range based on a comparison of the two images (orcolorspace data points). If the current image is equal to or within thepredetermined range of the reference image, then the inspection proceedsforward. If the current image is not within the predetermined range ascompared to the reference image, then the camera settings are adjustedto match the reference image.

In an effort to match the reference image, the camera settings may beadjusted to compensate for extra or a lack of light within the building.The camera 108 has various settings that may be adjusted including theexposure, gain, brightness, and/or color.

An electronic control unit 110B is configured to communicate directly tothe camera 108 to adjust said camera settings. A second image,specifically a second reference image, is taken using the adjustedcamera settings. The processor 112 then restarts the process where thesecond current image is compared to the reference image.

Data regarding the comparison of the current image to the referenceimage is immediately sent to a data storage unit or cloud resulting inlearning of the system 200. Data sent to the cloud may include specificsettings with respect to a certain time of day, weather data, generallighting within the building determined by a sensor and/or time of yearbased on daylight saving or total available light during the day. Thisdata may be used and incorporated with the histogram matching 202 and/orused with the learning 204.

The machine learning 204 approach uses a plurality of data from varioussources to determine the appropriate camera settings prior to taking afirst reference image. A cloud or other data storage unit stores aplurality of data relating to prior camera settings, lighting, weather,time of day, time of year and/or third party data. The processor 112generates a model image in accordance with said enumerated data. Theprocessor 112 then determines if the model image is within thepredetermined range as compared to the reference image. This process isconducted without first taking a current image.

The processor 112 creates a theoretical model image to compare to thestandard reference image. If the processor 112, based on the HSVcomparison, determines that the model image is not within thepredetermined range, then the camera settings of the camera 108 areadjusted prior to taking the first image.

An electronic control unit 110A controls the exposure, gain, brightness,and/or color of the camera 108 prior to taking the first image. Theimage is then take by the camera 108 of the object 106 to be inspected.The current image created based off of the learning approach is thenprocessed through the histogram matching 202, if time allows. Theprocess continues by comparing the current reference image based off ofthe machine learning 204 to the reference image to determine if the HSVcolorspace is within the predetermined range.

The machine learning 204 also uses the HSV colorspace to build thestatistical model by modifying the gain and exposure settings on thecamera 108 and extracting various features of the image. The machinelearning also uses CILAB color space (also referred to as LAB), such asshown below. LAB expresses color as three values: L* for the lightnessfrom black (0) to white (100), a* from green (−) to red (+), and b* fromblue (−) to yellow (+). LAB is an alternative means to represent animage, similar to RGB or HSV, and is arguably a closer representation asto how human perceive color. LAB keeps luminosity and color separate.The L channel captures the luminosity while a and b capture the color: acovers green to magenta and b does blue to yellow. By throwing inseveral colorspaces, especially in the machine learning side, sometimesthe algorithms pick up on details that aren't there in the RGB or HSVrepresentation. The features to be compared are enumerated below inTable 1:

TABLE 1 Features used in the machine learning algorithm Feature Mean ofthe RGB Blue plane Mean of the RGB Green plane Mean of the RGB Red planeMean of the RGB plane Mean of the LAB L plane Mean of the LAB A planeMean of the LAB B plane Mean of the LAB plane Mean of the HSV Hue planeMean of the HSV Saturation plane Mean of the HSV Value plane Mean of theHSV plane Standard deviation of the RGB blue plane Standard deviation ofthe RGB green plane Standard deviation of the RGB red plane Standarddeviation of the HSV hue plane Standard deviation of the HSV saturationplane Standard deviation of the HSV value plane Standard deviation ofthe LAB L plane Standard deviation of the LAB A plane Standard deviationof the LAB B plane Variation of the RGB blue plane Variation of the RGBgreen plane Variation of the RGB red plane Variation of the HSV H planeVariation of the HSV V plane Variation of the HSV V plane Variation ofthe LAB L plane Variation of the LAB A plane Variation of the LAB Bplane

Table 1 above enumerates a plurality of filters that are compared duringthe comparison step. The filters are compared or overlaid to determineif there are differences between them.

In the machine learning 204 configuration, the system requires aninitial training phase where the algorithm controls the gain andexposure settings on the camera and extracts the required features. Thefeatures are then fed into the machine learning algorithm where theplurality of data is stored in order to build an appropriate model thatdetails the given camera settings based on what the current image willlook like. When the algorithm is running, the algorithm takes as inputthe reference image and the current image from the camera, uses themodel to determine at what gain an exposure settings the input imagelooks like and then changes the gain and exposure settings so that thereference image and the current image will match.

FIG. 2 generally depicts one embodiment of a system 100 including aconveyor 102 having a plurality of rollers 104 to move the objects 106forward for inspection. The inspection system generally includes acamera 108 in communication with an electronic control unit (ECU). Aprocessor 112 is in communication with both the ECU 110 and the camera108.

The functionality and steps as described above is applied to the system100 as illustrated in FIG. 2. In this embodiment, the camera 108 takesthe reference image and the current image from an upward angle. Inalternative embodiments, such as illustrated in FIG. 3, the camera takesthe reference image and/or the current image from a side angle. Infurther alternative embodiments, the camera may be taken at differentangles or multiple cameras may be provided to enhance accuracy of thecurrent image in the reference image taken.

In further embodiments, the software and corresponding camera 108 isused to mask out areas on the reference image that may impact thequality of the current image. Specifically, the software and/or camera108 may be used to mask out areas of the reference image that should notbe part of the calculation due to the color matching. Masking is helpfulin situations where there is, for example, a shiny or metallic portionthat could negatively impact the functioning of the comparison of thereference image to the current image.

In other embodiments, the machine learning 204 may also be used to takein multiple reference images over the course of a day to capture allpossible lighting conditions that could be encountered to make sure thepart is visible. In this embodiment, a plurality of reference images aretaken throughout the day to compensate for changes in ambient light,habits of opening doors and windows, and/or the time of year andcorresponding light associated with that time of year. In thisembodiment, a plurality of reference images could be rotated throughoutthe day depending on the various conditions.

Alternatively, the reference image newly taken, by example every fewhours, would replace the prior reference image taken and currently used.

In additional embodiments, the machine learning 204 is used as anincremental learning system where the underlying model is updatable asnew instances of data are uncovered. In this embodiment, various piecesof data such as prior camera settings, lighting, weather, time of day,time of year, and/or third party data is constantly used to update thecomparison system conducted by the processor 112. This method wouldallow for constant learning and updating providing for the best possibleand accurate comparison between a reference image and a current image,or between a model image and a reference image.

It is noted that the terms “substantially” and “about” may be utilizedherein to represent the inherent degree of uncertainty that may beattributed to any quantitative comparison, value, measurement, or otherrepresentation. These terms are also utilized herein to represent thedegree by which a quantitative representation may vary from a statedreference without resulting in a change in the basic function of thesubject matter at issue.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

What is claimed is:
 1. An automated inspection process for inspectingimaged on a conveyor belt of an assembly line using machine learningcomprising the steps of: storing and processing a plurality of datarelating to a camera and/or image processing; generating a theoreticalmodel image of an item to be inspected prior to taking a first imageusing the plurality of data relating to the camera and/or imageprocessing; determining if the model image is within a predeterminedrange; adjusting the camera settings to match the model image; taking areference image based on the model image; taking a current image of theitem on the conveyor belt of an assembly line; and comparing the currentimage to the reference image to determine if the current image is withina predetermined range.
 2. The automated inspection process of claim 1wherein the plurality of data includes prior camera setting data.
 3. Theautomated inspection process of claim 1 wherein the plurality of dataincludes weather data.
 4. The automated inspection process of claim 1wherein the plurality of data includes lighting data.
 5. The automatedinspection process of claim 1 wherein the plurality of data includestime of day data.
 6. The automated inspection process of claim 1 whereinthe plurality of data includes third party data.